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ABSTRACT 


This paper presents an integrated system for emotion detection using facial 
detection and recognition, we have taken into account the fact that emotions 
are most widely represented with eyes and mouth expressions. In this 
research effort, we implement a general convolutional neural network (CNN) 
building framework for designing real-time CNNs. We validate our models by 
creating a real-time vision system that accomplishes the tasks of face 
detection, emotion classification, and generating the content according to the 
emotion or mood of the person simultaneously in one blended step using our 
proposed CNN architecture. 

Our proposed model consisted of modules such as image processing. Feature 
extraction, feature classification, and recommendation process. The images 
used in the experiment are pre-processed with various image processing 
methods like canny edge detection, histogram equalization, fit ellipse, andFER 
dataset is mediated for conducting the experiments. With a trained profile that 
can be updated flexibly, a user can detect his/her behavior on a real-time 
basis. It utilizes the state of the art of face detection and recognition 
algorithms. 

KEYWORDS: classification and recognition, Convolutional Neural Networks, 
Feature Extraction 


I. INTRODUCTION 

Human emotions and its detection and recognition play a 
very vital role in applications intended for handicapped 
persons or robotic intelligence or persons with difficulty in 
recognizing emotions or for developing intelligent Graphical 
Application. To detect the facial expressions on the human 
face his face must be detected and the face location and 
position must be known. Hence in most cases, emotion 
detection algorithms start with face detection, taking into 
account the fact that face emotions are mostly depicted in 
eyes and mouth. 

There are two mutually benefited subtasks in any facial 
detection and recognition task that include Detection and 
Extraction of the image features, facial expression 
recognition system detects a face within the captured image 
and then it extracts features from the potential face for the 
recognition task. There are many different types of emotions 
that influence how we live and interact with others. At times, 
it may seem like we are ruled by these emotions, we intend 
to find this type of emotion like sadness, anger, happiness, 
disguise, etc. 

Furthermore, CNNs are used as black-boxes and often their 
learned features remain hidden, making it complicated to 
establish a balance between their classification accuracy and 
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unnecessary parameters Therefore, we implemented a real¬ 
time visualization. 

II. Literature Review 

lliasMaglogiannis, Demosthenes Vouyioukas, and Chris 
Aggelopoulospresented [1] presented an emotion detection 
and recognition system for color images. The proposed 
system detects the skin color using MRFs estimation in 
YCbCr space, overcoming the difficulty ofvarious skin 
regions detection under different lighting conditions, 
emotion recognition then follows, based on the construction 
of face, eye, and mouth maps. 

Octavio Arriaga and Matias Valdenegro [2] have proposed 
and tested general building designs for creating real-time 
CNNs. Their architecture has been systematically built to 
reduce the number of parameters, by eliminating the fully 
connected layers and by reducing the number of parameters 
in the remaining convolutional layers via depth-wise 
separable convolutions. 

Bee Theng Lau [3] developed to help to detect the emotions 
of the disabled who could not make a speech and also 
physical movements easily. These people tend to be seated at 
one fixed location or wheelchair and require assistance in 
most of the physical movements, the detection and 
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recognition rate is up to 89%, this is sufficient for assisting 
the disabled and their family members or caretakers to use 
in their daily life. 

Wenlu Yang, Christophe Marsala [4] Investigates the 
psychophysiological response to events on an affective 
database in a game context. We address some common 
challenges for a physiological-based affective model such as 
segmentation, normalization, or relevant signals. For 
emotion detection task, we show that high arousal emotions 
are more detectable than the low arousal ones. 



Figl. Samples of FER 2013 Dataset 


A. Converting Images into Grayscale Format 

Grayscale is a range of monochromatic shades from black to 
white. Therefore, a grayscale image contains only shades of 
gray color and no color. While digital images can be saved as 
grayscale images, even color images contain some grayscale 
information. This is because each pixel has a certain 
brightness or intensity value despite its color, intensity 
values which can be measured on a scale from black to 
white. Black means zero intensity and White means high 
intensity, grayscale images are less complex and having an 
only single layer for processing and thus making it efficient 
for processing. 

B. Histogram equalization 

Histogram Equalization is a technique in image processing 
for adjusting image intensities to enhance contrast. Some of 
our images are blurry, dark and fragile we need to enhance 
those images by using Open CV function. For equalizing the 
histogram, we need to compute the histogram of the image 
and then normalize it into a probability distribution. For 
normalization, we just need to divide the frequency of each 
pixel intensity value by the total number of pixels present in 
the image. This is equal to the resolution of the image. The 
equalization process makes sure that the resulting histogram 
should be flat. 


III. Emotion Recognition 

The process of emotion detection system is to develop a 
system that can recognize the faces under different viewing 
angles in real time using video in an uncontrolled 
environment. The only problem persists while doing this is, 
we need to update the output in real time. 

Facial detection is important step in emotion detection. The 
first thing to do in any machine learning task is to collect the 
data. What we need are thousands of images with labelled 
facial expressions. The public FER dataset is a great starting 
point for that. We imported the FER 2013 dataset for the 
emotion detection step. 



Fig2. Our Proposed System Architecture 


Before we train an emotion detector, we first must be able to 
locate the faces within each image. Since the goal is to 
classify emotions from facial expressions, it makes sense to 
only keep the faces we find and ignore all the irrelevant 
things. 


C. Neural Network 

Implementation database is categorized as the 
Preprocessing phase. Learning phase, validation and testing 
phase. This has been carried outusing Keras and its functions 
which significantly reduces the time for neural network 
training, validation and testing. Keras is a high-level neural 
networks API, written in Python and capable of running on 
top of Tensor Flow, CNTK, or Theano. 

Our Convolutional neural network model consisted of layers 
of Conv2d and Maxpooling parameters. Keras Conv2Dis a 
2D Convolution Layer, this layer creates a convolution kernel 
that is a wind with layers input which helps produce a tensor 
of outputs. Max-pooling reduces the dimensionality of 
images by reducing the number of pixels in the output from 
the previous convolutional layer and it also reduces the 
problem of over fitting. 


Emotion classifier Input 


Fig3. Greyscale Conversion 

D. Recommendation System 

Recommender systems are algorithms that support human 
decision making by reducing the amount of available options 
to choose from. Understanding the factors that influence 
decision making is important for designing better 
recommender systems. Emotions and personality traits are 
psychological constructs that are linked to decision making 
and are used to improve recommendations. 

The input is acquired in real-time. The hidden markovmodel 
classification are used for processing the acquired images. 
The frames that are obtained are considered in all frames 
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and all pixel formats for the purpose of emotion 
classification. The value of each landmark in the face is 
calculated and is stored for future use. The efficiency of 
classifier is about 90-95%. 

The emotions are assigned for every content shown. When 
the emotion is transferred the respective content will 
displays. There are four emotions that can be used and the 
emotions are happy, anger, sad, surprise. When the happy 
emotion is recognized the content that are assigned for that 
particular emotion are played and the same happens with 
the other emotions as well that is it the content is displayed 
for the emotions detected respectively. 

Hybrid Recommendations 





Fig 4 Recommendation Module 



Reco 


IV. Result 

We validated this model in the FER-2013 dataset. This 
dataset contains 35,887 grayscale images where each image 
belongs to one of the following classes [“angry", “disgust", 
“fear", “happy", “sad", “surprise", “neutral"]. 

Table No.2 Sequential Model Performance 


Epochs 

Loss 

Accuracy 

1 

1.4103 

0.4147 

2 

1.1550 

0.5257 

3 

1.0292 

0.5817 

4 

0.9121 

0.6352 

5 

0.8040 

0.6867 

6 

0.6949 

0.7329 

7 

0.5645 

0.7840 

8 

0.4347 

0.8373 

9 

0.3135 

0.8875 

10 

0.2440 

0.9150 


The system was trained with training data set, and validated 
with validating data set to find the best network architecture 
and find the optimal solution. Our initial model achieved an 
accuracy of 65% in this dataset. 


Model Accuracy 



Model Loss 



V. Conclusion 

This paper has described an approach to detect and 
recognize face with the CNN and show contents like music, 
blog, news according to the emotion of the person and it 
describes its overall performance. The algorithm proposed 
has three main stages Preprocessing, neural network 
classification and testing. Our analysis shows that the 
performance of Neural network data with a precise accuracy 
rate of 65%. This could significantly use to identify people 
suffering from depression, anxiety etc. and at the same time 
help them too. If user is often angry, program will suggest 
him tips for anger management. Future work also would 
investigate the use of a better complementing feature for 
real-time classification and recognition and showing the 
results live. 
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